Naive Application of Permutation Testing Leads to Inflated Type I Error Rates

Naive application of permutation testing leads to inflated type I error rates.

Failure to account for family structure within populations or in complex mating designs via uninformed applications of permutation testing will lead to inflated type I error rates. Careful consideration of the design factors is essential since some situations allow several valid permutation strategies, and the choice that maximizes statistical power will not always be intuitive.

Multiple testing. Part I. Single-step procedures for control of general type I error rates.

The present article proposes general single-step multiple testing procedures for controlling Type I error rates defined as arbitrary parameters of the distribution of the number of Type I errors, such as the generalized family-wise error rate. A key feature of our approach is the test statistics null distribution (rather than data generating null distribution) used to derive cut-offs (i.e., rej...

Conceptual Distinction between the Critical p Value and the Type I Error Rate in Permutation Testing

Type I Error Rates and Power of Multiple Hypothesis Testing Procedures in Factorial ANOVA

There are numerous General Linear Model (GLM) statistical designs that may require multiple hypothesis testing (MHT) procedures that control the Type I error inflation that occurs with multiple tests. This study investigated familywise error rates (FWER) and statistical power rates of several alpha-adjustment MHT procedures in factorial ANOVA, but results apply broadly to GLM procedures. Of fou...

A Response to Anderson's (2013) Conceptual Distinction between the Critical p value and Type I Error Rate in Permutation Testing